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ABSTRACT 



The Politecnico di Torino Joint Committee for Teaching (CPD) 
has set up an observation system consisting of: a) extended course 
presentations, b) statistics on examinations, and c) student questionnaires. 
At the end of each academic year, the CPD publishes volumes containing 
statistics on examinations, where each course is described by a cumulative 
curve showing the ixu:;i>er of students passing the exam with time (examination 
sessions); each course is followed for four years. The CPD has also developed 
a system of questionnaires on teaching quality to be distributed to the 
students. In these questionnaires, students provide information concerning 
their attendance rates, work required by the instructor, work load for the 
semester, classrooms, lecture hall and facilities, teaching abilities of the 
professor and teaching assistants, and timetables. Questionnaires and 
examination statistics provide the raw data or "descriptors" used to obtain a 
system of teaching quality "indicators, " which is published yearly and made 
available to the academic authorities at many levels. (Author/ASK) 
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Abstract - The CPD has set up an observation system 
consisting of a) Extended course presentations, b) Statistics 
on examinations, and c) Student questionnaires. At the end 
of each academic year, the CPD publishes volumes 
containing statistics on examinations, where each course is 
described by a cumulative curve showing the number of 
students passing the exam with time (examination sessions); 
each course is followed for four years. The CPD has also 
developed a system of questionnaires on teaching quality to 
be distributed to the students. In these questionnaires, 
students provide information concerning their attendance 
rates, work required by the instructor, work load for the 
semester, classrooms, lecture hall and facilities, teaching 
abilities of the professor and teaching assistants, and 
timetables. Questionnaires and examination statistics 
provide the raw data or " descriptors " used to obtain a 
system of teaching quality "indicators", which is published 
yearly and made available to the academic authorities at 
many levels. 

Introduction 

The quality of university teaching depends first of all on the 
curriculum, which must be designed to meet the student's 
future needs as a self-employed professional or as a private 
or public sector employee, as well as the need to keep pace 
with change. The curriculum for a degree course must thus 
state the type of cultural background to be transmitted to the 
student, and the skills he must show he has mastered. This 
involves the non-generic preparation of an educational 
program, which takes the form of the academic institution's 
"Mission Statement" and consists of a "Reference Frame" 
for evaluation [1] integrated by appropriate thought on the 
quality of engineering education, the type of learning 
required of the student, and the assessment methods [2,3]. 

The curriculum thus crystallises the intentions 
established through a feedback loop linking the university's 
governing bodies and the "customers". Proposed changes to 
the upper feedback loop will be relatively slow: they must 
take general political and social needs into account as well 
as those of the "customers" (including students themselves). 

They must be only partially constrained by the 
resources available to the university, though obviously these 
constraints are very strong. Below the curriculum level, a 
number of subsidiary feedback loops can be set up to 
determine whether the procedures meet objectives, and 
whether they meet them efficiently. 




Since 1993, the Politecnico di Torino Joint Committee 
for Teaching (CPD) has been working with these subsidiary 
feedback loops. The CPD is an independent monitoring 
organisation, but also submits proposals (at an advisory 
level) to the University's governing bodies. Politecnico di 
Torino was the first university in Italy to establish a 
permanent body for observing and evaluating teaching 
problems, teaching policies, and material means and 
services connected with teaching. 

Several universities in Italy have set up committees 
which deal with certain aspects of teaching. Only the 
Politecnico di Torino, however, has a CPD, characterised by 
the following features: 

• It is a permanent body. 

• It consists of an equal number of professors and 
students (8+8). 

• It has the statutory task of organising a permanent 
observatory on the functionality of teaching activities. 

• It has the statutory task of evaluating the effectiveness 
of the policies implemented by the bodies which deal 
with teaching. 

To assist in setting up the "subsidiary feedback loops" 
in a spirit of continuing self-evaluation, "indicators" are 
provided for: 

• Rector and Academic Senate, 

• Faculties, 

• DCB-Degree Course Boards, 

• individual instructors. 

They are described at the end of this contribution. 
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Observation Activities 

From 1993 to 1997, the CPD formulated, employed and 
perfected observation tools capable of producing the 
"descriptors”, whose definition was in itself a demanding 
task. To this end, the CPD has launched a co-ordinated 
system of integrated observation tools: 

• Extended Course Presentations 

• Statistics on Examinations 

• Student Questionnaires on the quality of teaching 

From the early development of these tools, the CPD has 
worked to: 

• Deal with problems with a global vision, while 
suggesting particular solutions for each tool through 
experimentation and evaluation of its feasibility; 

• Inform the university bodies of the actions undertaken, 
gathering their opinions. 

Every effort has been made to make it clear to all 
concerned that each of these observational tools is part of a 
comprehensive, single design. An objective difficulty was 
that a portion of the faculty perceived these tools in a 
fragmentary manner, or simply did not share some or all of 
their aims. 

Extended course presentations were proposed by the 
CPD with a dual purpose: 

• To give students a better insight into the course so that 
they can organise their semester work, arrange their 
choices, plan their study and know inadvance what is 
expected from them. 

• To give those responsible for running the Degree 
Courses means for monitoring course contents, the 
extent of their treatment, requirements andworkload, 
examination techniques, and the abilities that the 
student is expected to develop. 

It is clear that the first point is instrumental in 
establishing a sort of contract between teacher and student, 
where both act in a context of mutual exchange so that final 
evaluation can be based on comparison with the objectives 
stated in advance. It is also clear that the second point gives 
the governing bodies (particularly at the DCB levels) a tool 
for monitoring the system so that they can make timely 
corrections in accordance with the institution's policies. 
Moreover, it provides them with a more factual basis for 
evaluating the results from the other two tools. 

The Extended Course Presentation is not only a useful 
document for students to help them plan their activities 
during the scholastic year, but is also a declaration of intent 
by teachers which helps to co-ordinate those teaching 
disciplines that serve a common didactic aim. It also serves 



as a means of comparison with the results of the student 
Questionnaire. Finally, the Presentation is considered 
indispensable for adopting the ECTS within the Socrates 
project. 

After a year or so of discussion, it was decided to 
incorporate most of the features suggested by the CPD in 
the course presentations. Following collaborative 
discussions between the deans, the DCB and the Student 
Services, the Student Handbook now being printed has been 
doubled to include a minimum of two pages per course. 

Student Questionnaires on Teaching Quality 

Student questionnaires on teaching quality are strongly 
oriented towards evaluating individual teaching as a part of 
an organized system. Questionnaires consist of two parts: 
one is an OCR form containing the questions whose 
answers are processed by the CPD and are used as the basis 
for their evaluation. 

The second is a free-form sheet for "personal 
comments", which are received directly by the individual 
professors, are strictly confidential, and act as a 
communication vehicle between class and professor. These 
sheets are anonymous, which spurred initial opposition from 
around 15 to 20 % of the professors. 

The form (version 1997/98)contains several questions 
grouped as follows: 

• Part A: Six questions on the student, his position, and 
prior education. 

• Part B: Three questions on the attendance rates for 
lessons and practice sessions, and the examination 
strategy. 

• Part C: Four questions on the course organisation, 
lecture rooms, material resources. 

• Part D: Three questions on the context of the course 
(weight of parallel courses, general work load, 
timetables). 

• Part E: Twelve questions on the professor, his 
performance, his ability to communicate, his 
commitment. 

• Part F: Six questions on the teaching assistants. 

• Part G: Three control questions (global evaluation, 
motivation). 

Students are asked to assign ratings by case or by value 
on a 1 to 5 scale (1 = worst, 5 = best). 

This set of questions gives a thorough view of how the 
student evaluates the course and the professor (and his 
assistants) in relation with the others and with the facilities 
made available by the institution. It requires complex 
processing which is performed at the end of each semester 
and again following departmental checks at the end of every 
academic year. 
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The student questionnaires on teaching quality 
underwent a long process of revision and experimentation. 
As a result of much reflection regarding the organization of 
the questionnaires, the CPD decided on the following ways 
of distributing the results: 

• Each teacher receives all raw data pertinent to his/her 
courses 

• CPD members, Rector and deans have access to all raw 
data - "descriptors" 

• Department heads and chairmen of the DCBs have 
access to raw data concerning the teachers in their 
department / DCB 

After examining the questionnaires from the previous 
academic year, the CPD decided on two courses of action 
regarding those situations that required further control. 

Improper Use of the "Personal Comments" Form 

Many teachers, particularly in the first year courses, pointed 
out examples of an inappropriate use of the "personal 
comments" form. In many cases, the comments made on 
this form were of a non-constructive, personal nature. 
Behind the shield of anonymity, these comments were in 
some cases blatantly offensive. The CPD first of all wanted 
to discover the dimensions of this phenomenon, and to this 
end formulated a brief questionnaire. It was discovered that 
the phenomenon was not uniformly distributed, but 
concentrated primarily on certain cases, particularly in the 
first year courses. The CPD initially considered the 
possibility of not distributing the questionnaires for the first 
year courses. Nevertheless, the majority of teachers 
indicated a desire to continue to use the forms as a valid 
communication tool for improving teaching quality. The 
CPD introduced several changes, hoping to provide students 
with a better understanding of the type of response that is 
requested of them. 

Non-Distribution of Questionnaires 



A number of questionnaires was not distributed in the first 
year the programme was introduced. This was partly due to 
difficulties in starting up the CPD procedure, and partly to 
the position taken by a teachers’ organisation, which 
opposed the questionnaires. In addition, certain teachers 
failed to distribute the questionnaire and, furthermore, did 
not justify their reasons. The CPD therefore decided to give 
students the possibility of completing these questionnaires 
either at home or at an appropriately discreet location during 
the month of September. To this end, the students received a 
letter sent directly to their residential address. Student 
response to this initiative was extremely limited, and the 
CPD was able to carry out purely indicative processing, 
based on those few disciplines that registered a response 
above 20%. This experience is not to be repeated. 



Tab. I: Overall redemption rate A.Y. 1996/97 



School or 
Faculty 


N 


Nd 


Nr 


% 

Nr/Nd 


Architecture 


203 


26905 


7930 


29 


Diplomas 


355 


14504 


8195 


57 


Eng. Torino 
and Mondovi 


540 


55787 


23328 


42 


Eng. Vercelli 


86 


3747 


2271 


61 


Total 


1184 


100943 


41724 


41 



N = Total monitored courses, 

Nd ~ Total no. of questionnaires 

Nr = Number of redeemed questionnaires 

The operation questionnaire is now applied to all 
courses ofPolitecnico di Torino. 

Tables I and II provide statistics on the Student 
Questionnaires from Academic Year 1995/96, when the 
response appeared to have stabilized. 

Courses with less than 10 students are not required to 
distribute the questionnaires. However, they may ask to 
participate in the operation, in which case they are included 
in the statistics. 



Tab. II: participation rate of teaching staff 





Academic Year 1995/96 


Academic Year 1996/97 


School or faculty 


N 


Nd 


%Nnd/N 


N 


Nnd 


%Nnd/N 


Architecture 


185 


18 


10 


203 


21 


13 


Diplomas 


256 


13 


5 


355 


38 


11 


Eng. Torino/Mondovi 


523 


24 


5 


540 


25 


5 


Eng. Vercelli 


78 


0 


0 


86 


4 


5 


Total 


1042 


53 


5 


1184 


94 


8 



N = Total monitored courses, 

Nnd = Monitored courses where questionnaires were not distributed 
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Statistics on Examinations 

The statistics on exams and their temporal sequence are the 
first objectives achieved by the CPD. The collaboration of 
the NPS, the S1A, the Student Service of the Politecnico and 
the deans were all vital in reaching this goal. From the 
beginning, the CPD demonstrated the need to make data 
regarding the students' educational progress available at 
regular intervals. In connection with similar needs expressed 
by the university teaching bodies, it contributed to the 
formation of a centralised information service under the 
responsibility of the S1A, which gathers the data and makes 
it compatible with data arriving from the Student Services 
and the deans. Procedures for gathering and processing the 
data have been standardised and by now are almost entirely 
error- free. The results are sent to the appropriate department 
heads, who forward them to the teachers for any necessary 
corrections and observations. At the end of this validating 
procedure, the information is sent to the Rector, the deans, 
the CSCD, the academic authorities and the Student 
Services. Copies are available at the offices of the CPD and 
in the university libraries for public scrutiny. 



The observed "universe" is the cohort of students 
enrolled for the given course in the given academic year. 
They are followed for four years in order to highlight when 
the examination is passed. A cumulative curve is then 
obtained, which should ideally tend to 100% with features 
which vary from course to course. The basic parameters 
which are extracted from these data (% of pass at the end of 
the semester, % of pass at the end of the academic year) are 
correlated with those from the questionnaires and form the 
basis for identifying obstacles to student careers. 

The following figure shows an example of examination 
pass curves. Note that 1, 11, 111 are the examination sessions 
available to the students during the year (January-February, 
June-July, September). The figure shows four curves 
because follow-up lasts four years, and this course was 
given by the same professor over that time, dropout 
students, i.e. those who have abandoned the course 
(transfers or exit from university) should be subtracted from 
the initially enrolled number. 

This is of interest to the first year courses, which are 
affected by a higher dropout rate. However, in order to 
preserve the shape of curves over the years concerned , this 
reasonable correction is not applied, though it is visually 
suggested by the fact that the asymptote is less than 100%. 




Fig. 1 : Typical record for course X given by professor Y over four years left: cumul. pass curves, right: distribution of marks 

(earliest avail. Acad. Year) 



Indicators 

Raw data or "descriptors" from questionnaires and 
examination statistics are processed to obtain "indicators" of 
teaching performance. Their development was and remains 
the object of much debate and controversy. They try to 
satisfy conflicting requirements: to give a correct and 
exhaustive portrait of the situation while conveying 
information simply and quickly. Homogeneous groups of 
descriptors are combined together into a reduced number of 
indicators ordered by columns. Each column describes a 
course and its teacher/instructors; it is transmitted to the 



teacher. Columns are grouped in "sheets", each relating to a 
given year of a given degree course. The collection of sheets 
for the same degree course (three for Diploma, five for 
"Laurea") are transmitted primarily to the chairman of its 
DCB. The entire collection is made available to the 
academic authorities; at the moment it is not disclosed to the 
public. 

A sample sheet is shown in figure 2. Each indicator is 
ranked in a value scale: 1: Very poor, 2: Poor, 3: 
Acceptable, 4: Good, 5: Very good. In order to facilitate 
quick inspection, a system of colours is adopted: black - 
possible problem area, grey - average values, white - better 
than average. 
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The sheet is complemented by a table showing the 
distribution of students according to number of exams 
behind and number of years behind. All indicators of the 
sheet concern only "R-regular" students; however, this table 



also contains data on "F-fuori corso" students, i.e. students 
enrolled on a different basis due to excessive examinations 
behind. A bar diagram shows the distribution of "regular" 
students by number of exams behind. 



Fig. 2:sample sheet of indicators for "XX Degree Course", III year, semesters I and II 



XX Degree Course 






III Year 






Academic Year 1995/96 




Semester 1 






Semester 2 




% of passed exams, end 
semester 


68 


5 


66 


29 


43 


5 


% of passed exams, end Acad. 
Year 


85 


12 


73 


48 


50 


7 


% of lessons attended 


94 


93 


95 


88 


97 


93 


% of tutorials attended 


94 


93 


93 


96 


96 


71 


Sufficiency of tutors' number 


4 


3,6 


4,2 


4,4 


4 


4,6 


Adequacy of classroom space 


3,8 


4,6 


4 


3,7 


4,6 


4,6 


Total semester workload 


3,5 


2,7 


3,1 


1,2 


3,4 


3,4 


Workload of this course 


4,1 


2,1 


2,3 


1,6 


3,1 


2,6 


Rationality of timetable 


4,7 


4,5 


4,2 


3,1 


4,3 


3,8 


Overall organization of the 
course 


4 


3,6 


4,3 


3,2 


4,5 


3,7 


Lecturing effectiveness 


3,8 


2,9 


4,3 


2,1 


4,5 


3,4 


Teacher’s commitment and 
availability 


4,3 


3,7 


4,6 


3,7 


4,7 


3,9 


Tutorial instructors’ 
effectiveness 


3,9 


3,4 


4,5 


4,1 


4,3 


3,5 


Effect, part-time student 
assistants 


NA 


NA 


3,9 


1,9 


4,2 


3,1 


Student interest in the subject 


3,6 


3,8 


4,2 


2,3 


4,4 


3,4 


Student estimate of course 
relevance 


3,2 


4,2 


4,4 


2,1 


4,6 


3,6 


Course title 


Title of 


Title of 


Title of 


Title of 


Title of 


Title of 




course 1 


course 2 


course 3 


course 4 


course 5 


course 6 


Course code 


TOxyOO 


TpqmOO 


TxzOOO 


TOxtOOO 


TabaOO 


TOadfO 


Teacher name 


Prof. 1 


Prof. 2 


Prof. 3 


Prof 4 


Prof. 5 


Prof. 6 


Main reasons for low lesson 
attendance 


- 


- 


- 


- 




■ 


Main reason for low tutorial 
attendance 


- 


- 


- 


- 


- 


A(3) 


% redeemed 
questionnaires 


- 


29 


71 


93 


86 


29 


No. enrolled students 


48 (48) 


49 (49) 


48 (48) 


48 (48) 


48 (48) 


48 (48) 


Faculty / City identifier 


XY 


XY 


XY 


XY 


XY 


XY 
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2 


3 


4 


6 


7 


10 


>10 


Tot. 






% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 






Reg. 


FC. 


Reg. 


FC. 


Reg.jFC. 


Reg. 


FC. 


Reg. 


FC. 


Reg. 


FC. 


Reg. 


FC. 


3 

3 


o i 


4,2 








t 




, 












4,2 




if 3 


18,8 




2,1 




f 




, 












20,8 




w 

.Q 


ST 


3173 




873 


7TJ 


2j 




, 












41,7 


11,1 


1 


7 10 






6,3 


33,3 


4,2 


33,3 


2,1^ 


22,2 










31,3 


88,9~ 


\ f V 

K 


>io~ 


, 




2,1 




, 




, 




9 








2,1 




Tot. 


70,8 




18,8 


44,4 


6,3 


33,3 


2,1 


22,2 


t 


A 






100, 


100, 



% Distribution of 

Ni Regular (Reg) students and of 

Nf Fuori Corso (FC) students by years of delay and exams behind 




Distribution of Ni regular students by number of examinations behind. 



Ni=48 

Dropouts = 0 
Fuori Corso (FC)= 9 
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